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Use of Measures of Potential and Motivation in a Promotion Examination 
- • . From Laborer-Type Positions to Gardener-Trainee 

Park Service 

PART ONE - INTRODUCTION 



I . PURPOSE 



We are making use of new measures of potential and motivation to evaluate 
applicants in the elements of positions to which they seek advancement. 

The purpose of this report is to analyze results of the second of our 
cooperative studies with agencies in the use of these measures, (The 
firsb study is reported in Report on New Tests of Reading, Math em a 1 1 c aj. 
Abilities, and Mechanical Information for Apprentice and Trainee Position s, 
September 19&3 , ) 



In this examination, applicants were evaluated on each element by usxng 
combinations of programmed instruction, aptitude tests, interest and 
motivation checklists, special supervisory ratings and information in 
personnel folders* 



The following summary of the promotion examination, prepared by Roy S« 
Ference, Bureau of Training, U» S, Civil Service Commission, appeared 
in the Federal Trainer, Vol. 1, No. 1, August 1969' 



Interior's National Capitol Region, National Park , 
Service, the second pioneer in this new approach, 
designed a program to train gardeners and tree workers. 
Using the new selection techniques stressing potential 
and motivation, 23 former laborers are now learning a 
trade that will pay them $900 to $l£00 more per year. 
All had been laborers all their lives; 5 were liO or 
older; b had no high school; and 19 are minorities. 

In the classroom they are learning the* theory and 
principles of plant growth and on the job they are 
following the seasons in applying that knowledge while 
beautifying our capitol. The 'greatest feedback from 
Parks' program is that ‘the technique of measuring a 
man's potential and interests in a-, trade, rather thai 
his previous experience and achievement, is a good, if 
not better, predictor. of how well he will do. Of 
great interest is that the few men who have left the 
gardener program were indeed at the lower scale in 
terms of total interests and motivation for the trade. 



The promotion examination, was conducted in the agency by Mr, Edward Jylka, 
who developed the crediting plan and supervisory rating forms, instructed 
the rating panel in job element procedures and furnished the data for the 
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present analysis. The training program itself was conducted by Mr, James 
Lindsay of the agency . 



II. CREDITING PLAN 

The crediting plan was based on the following elements, as given in 
Handbook X-118C for jobs like Gardener-Trainee, where ability to learn 
and advance is necessary: Reliability and Dependability, Jod Aptitude 

and Interest (so worded, instead of Shop Aptitude .and Interest in ^ 
the standard since it is not in a shop). Following Directions, Ability 
to Work as Members of a team and Dexterity and Safety. 



As an example of the way in which various kinds of evidence 'were used 
in’ the crediting plan, following is the crediting plan for the sere sn- 
out element. Reliability and Dependability. Note that in the job 
element system^ the sereen-out element is a summary ox all the elements 
in the standard* In the present instance* Reliability and Dependabili' y 
means the extent to which the employee can be depended upon to do 
the job. The standard provides that potential ability is required in 
particular in the element — Job Aptitude and Interest: 



Element 87 (Reliability and Dependability ) 
Evidence of Ab ility 



Point Values 



Superior Ability 

Able to render continuous and dependable 
service in situations relevant to tne kind 
of position applied for (for example, a 
person with 2 years or more in his most 
recent and related positions .with no 
evidence of poor attendance or other signs 
of unreliability ) . 

Satisfactory ability 

Demonstrated dependability in situations 
pertinent to the job applied for, with no 
negative evidence (or, otherwise intermediate^-' 
between 2 and U points) . 



Acceptable ability 

No seriously derogatory information. 

This is the scr een-out element . Note that 
meeting this element requires showing 
sufficient Job Aptitude and Interest. 

Point values below 2 disqualify the applicant. 
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Reliability and Dependability ( Continued) 



Suggested sources for evaluative purposes can be t 

1. Reading test 

2. Ability and Interest Questionnaire Scores, and answer 
to the questionnaire 

3. Supervisor's appraisal 

Ii. Employee's Official Personnel Folder 

5. Demonstrated performance by the Tree Worker Helper- 
Trainee applicant 

6, Any other factors that the Evaluation Committee feels 
appropriate 

Important Mote ; 

Written tests may not be used: 

- as sole indicator of ability in more than two elements 

- as a sole indicator of ability in the screen- out element. 

Ill „ SUP ERVISORY APPRAISAL OF EMPLOYEE FOR PROMOTION 

An appraisal form was developed to secure information from supervisors. 
Each employee was rated by his own supervisor and could, if he wished, 
ask to have a second supervisor rate him. The supervisory appraisal 
included the following categories; Reliability and Dependability, 
Following Directions, Ability to Work as a Member of a Crew, Safety and 
Dexterity} and Capabilities or Potential. 

a. Reliability and Dependability 

For the purpose of the supervisory evaluation, reliability and 
dependability in the appraisal applied only to past employment. 
This would, therefore, be only one part of the appraisal that 
would affect rating of the screen-out element even though.it had 
the same title. Other categories, particularly capabilities 
or potential, were also considered in rating the screen-out 
element. 

The category. Reliability and Dependability, was rated in three 
phases. The first phase was failing to report for work without 
first obtaining official leave, with categories of - — 

; Never 

No more than once in 6 months 
No more than once a month 
No more than once a week 
More often than once a week 



The second phase was frequency of leaving job before 
quitting tine without prior approval, with the same 
categories as above* 

The third phase was a judgment, "In your opinion, the 
applicant is — - 

Dependable and reliable; or 
Not dependable and reliable " 

b * Following Directions 

This was rated in the following categories; Often gets 
directions which change what he is doings and he does 
not have trouble following them; sometimes gets new 
directions and no . trouble * simple directions and no 
trouble j has to have simple directions repeated*’ 

c * Ability to Work as a Member of a Crew 

This was rated in terms of the following categories: 
Experience in cooperating where dangers are constant; 
experience in work needing close cooperation | experi- 
ence in cooperating in the regular run of work; no 
experience where he had to cooperate, 

d* Safety and Dexterity 

This was rated in terms of the following categories, not 
necessarily a continuous scale; Has observed and reported 
dangerous conditions; never had an accident, where he was 
at fault, with equipment on - job; no such accident in last 
2 years; more than one such accident in last 2 years, 

e , Capabilities or Potential 

This was rated in two phases. Firsts the supervisor checked 
whether the applicant had or did not have "the potential 
and capabilities to learn the knowledges and develop the 
skills required for target position," Then, the supervisor 
made a summary rating; 

Outstanding 

Excellent 

Satisfactory 

Unsatisfactory 
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XV, APTITUDE AND INTEREST BIAHK AND TESTS 
A- Aptitude and Interest Blank 

Use was made of a new kind of aptitude and interest blank that had been 
developed through the’ cooperative efforts of: 

Dr* Kenneth R, Brown - FMEDC, Standards Division 
Sally Ann Jones - PMRBC, Standards Division 
Ernest S. Priinoff - PMRDG, Standards Division 
Jack Me Court — BRE 
Thomas J. Portelance - BRE 

Behavioral evidences in past experience, training, or hobbies which would 
demonstrate aptitude and interest for Gardener-Trainee were collected* 

Most of the items were selected from 11 Questions on Your Trades Interest” 

(CSC Form 668), Additional items were added which had special relevance 
to gardening. 

For ease of competitors, items were grouped under the following categories: 
Use of Tools j Use Strength] Work Outdoors] Teamwork, in a Crew] Work With 
Plants or Trees] Mechanical Work] Careful and Reliable Work] Numbers and 
Arithmetic] 1 and Worked Under Special Conditions- For example, items under 
Use of Tools included: 

Cut wood with a saw 
File a piece of metal ■ 

Use a screw driver 

Items under Work With Plants or Trees included: . 

Rake leaves 
Pull weeds 
Water grass 

•Prepare earth for planting 
Items for Worked Under' Special Conditions included: 

Work where it 1 s damp 

Work in a narrow space, in a cramped body position 

Before the applicant filled out the blank, he listed his jobs on a nuiriDered 
sheet and his past training in mechanical, outdoors, gardening or arithmetic, 
on a lettered page. Then, as he began to fill in t he first category on the 
page, "Use of Tools, 11 he was told to write the Job Number for each job and 
the Course Letter for each training where he used or learned about tools. 

He was also to list any hobbies where he used tools. Then, for each evidence 
of use of tools, such as "Cut wood with a saw,” he was to mark one or more 
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of the letters—A* B* G* B=— to show his degree of ability* - - 

A, I have little or no training or experience in this 

B, I have had study or training in this 

C, I have done this in work or hobbies 

B. I feel I can do this well enough to make a living* 

without more training 

Then., he was to show his interest and willingness by circling L or B to 
show that he would - - 

L - Like doing it* or 
B - Dislike doing it 



and he was to circle ¥ or U to show that he would be * - 

W - Willing to do it* 'or 
U - Unwilling to do it 

The competitor followed the same procedure with the items in each of the 
categories. After the applicant completed all the items through the last 
category* "Worked Under Special Conditions* M he answered eight multiple 
choice questions to show his safety and attendance record# 

B. Written Tests 

1 * T est of potential for trades reading ' 

A reading test was prepared in accordance with "“the principles 
described in Report on New Tests of Reading* Mathematical" Abilities * 
and Mechanical Information for Apprentice and Trainee Positions " 
September 1968, The paragraphs concerned gardening# As indicated 
in the Crediting Plan for the element Reliability and Expandability* 
the reading test was considered as one evidence in the rating 0 f 
this element* It was also used as one evidence in evaluating Job 
Aptitude and Interest* and a minimum score was 

considered as one evidence that the applicant should receive at 
least one credit in Following Directions. 

2. Test of potential for trades ari thmetic 



A test of potential for using arithmetic in gardening was developed* 
based on the principles discussed in t he above mentioned report. 
Questions began at a lower level than ffilDse used for apprentice* 
and included problems in adding* subtracting* multiplication* 
dividing and use of fractions# The arithmetic test was used as one 
evidence in evaluating Job aptitude and interest. 
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3 * Tests 10 OA and 1QQE 

As provided in Handbook X-1180, Tests 100A (Gross Dexterity) and 
1Q0E (Following Oral Directions) were used as one indication or 
ability* 

Test 100A, Gross Dexterity, was used as one evidence in evaluating 
dexterity and safety. Test 10CE, Following Oral Directions, was 
used as one evidence in evaluating following directions. 

C, Performance Test of Tree- Climbing 

Some of the jobs demanded work in trees. To qualify for these jobs, 
applicants had to apply for and pass a tree- climbing performance test. 
The test was evaluated by two raters in terms of reaction to height, 
demonstrated agility, demonstrated stamina, general attitude during 
performance evaluation, examiner's opinion of employee 1 s reaction 
to lifting power saw, and other items the examiner might feel to be 
necessary. Each item was rated Very Satisfactory, Satisf actory, or 
Unsatisfactory. 

V* NUMBERS OF EXAMINEES AND OF ELIGIBLE3 



Fifty applicants completed all stages.^ Twenty- six of the applicants 
met the screen-out element as described in the rating schedule, but 
two of these did not receive an average of 2 credits in each element, 
as required in job element examining, ^ Thus, 2h passed the examination 
and each was selected for promotion. 



*-0ne additional applicant completed all stages, but withdrew 
voluntarily before being rated, ostensibly for unwillingness to 
accept a lower salary as trainee than his present salary 0 Two • 
additional applicants were rated by their own supervisors, but 
failed to complete aptitude and interest blanks or tests. 

^One feature of the examination was that applicants had been notified 
in advance that they would need to have a driver 1 s license, and 
possession of a driver's license was made part of the screen-out 
element. Five applicants were screened out because they did not 
have the license. 
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PART TWO - RELATIONSHIP BETWEEN THE 'VARIOUS EVIDENCES AND SELECTION 



x. THE EXTENT TO WHICH THE JOB-ELEMENT RA TI NG MAY BE 
CONSIDERED AS A CRITERION 



The job-element ratings were given by an evaluation committee,, the 
members of which took their task very seriously* especially because 
the new positions would, be involved, in the beautification program of 
the Nation's capitol. The committee included the head of the Park 
Service Regional Office. Each applicant's complete record was 
reviewed in detail* and members of the committee contacted super- 
visors and other agency officials who were knowledgeable about com- 
petitors whenever there were doubtful points. 

Since the various evidences in the examination were appropriately 
expected to have an effect on the ratings, relationships between 
particular evidences and final examination rating would be higher 
than if the ratings had been given independently of the particular 
evidences. Thus, any error in a particular evidence might be 
reflected by a corresponding error in the job-element ratirg. 

Nevertheless, since the job-element ratings represent a judgmental 
process performed by competent agency officials, the extent to which 
particular pieces of evidence are in conformity with the final judgment 
of the panel is an indication of the relative merit of the evidence. 

This is particularly important with respect to the aptitude and interest 
blank, which, as a self-report, tends to be looked upon with suspicion 
by officials who see it for the first time. 

II. SUPERVISOR'S RATINGS 

A, Reliability (See Table I, appendix) 

As indicated in Table I, the supervisory appraisal of the category 
Overall Ability had a reliability of .62 and appeared to be satisfactory 
as an indication of general satisfaction by a supervisor with the work 
of the applicant, at least in the laboring-type position. Table I also 
furnishes reliabilities of • the other ' categories in the supervisory 
, appraisal. " , 

B. Effect of Supervisory Ratings on Selection 

In the few cases where both supervisors indicated that the employee was 
not satisfactory in an important factor, he was not selected. These 
instances are shown in Table III in the appendix. Thus, the supervisory 
rating had an effect on the screen-out element as a negative influence. 
Although presence of positive words in the comments by the supervisor 
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would not have a positive relationship to selection, presence of 
negative words in comments by the supervisor correlated - . 25 with 
selection, being in the expected directions 

On the other hand, high supervisory ratings did not have in themselves 
an important positive effect on selection vs. non-selection. As indicated 
by the results at the beginning of Table III in the appendix, correla- 
tions between the supervisory ratings and selection (counted as l) vs. 
non-selection (counted as 0) were actually negative. Presence and number of 
positive words in the comments were actually negative with selection. 

0. Relation Between Applicant's Efforts in Getting Superviso ry 
Ratings and Selection 

Each applicant had. been permitted to either ask his own supervisor 
to fill in an appraisal form or have the personnel office ask the 
supervisor. 

An interesting finding was that 35 applicants who themselves asked 
their own supervisor to fill in an appraisal form and gave an addi- 
tional supervisor's name for an additional rating, had the greatest 
percent selected! U9 percent. 

Seven who themselves asked their oxfn supervisor but did not furnish 
another supervisor's name had 1±3 percent selected. Of eight who 
di d not ask their own supervisor but did name a different supervisor, 
only one was selected! 13 percent. These results may indicate that 
an applicant's wxllingne ss to ask for references is related to how 
well he is thought of in the organization, (Two applicants did not 
respond to a request to indicate their preferences as to how they 
, were to be rated, ) 

III, APTITUDE AMD INTEREST BLAMK AMD TESTS 

A. Effect of Aptitude and Interest Blank on Selection 

■ Self-rating in terms of degrees of ability on the various evidences 
correlated , 516 with selection. Self-rating in terms of interest and 
willingness correlated .01 h with selection,, (See Table III, appendix.) 

B, Written Tests 

Correlations with selection were ,701 for the test of potential for 
reading, .539 for the test of potential for shop arithmetic, . b9h for 
Test 100A, Gross Dexterity, and ,565 for Test 100E, Following Oral 
Directions, 
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IV. RELATION BETWEEN ITEMS IN THE PERSONNEL FOLDER AND SELECTION 

Years of schooling correlated .If? with selection. Age correlated 
-,U2 with selection. Length of experi ence in the Park Service 
correlated -.28 with selection, but only one applicant had less than 
lig years of experience, and he was not selected. Length of experience 
in the Federal service correlated -.21, and total length of experience 
correlated -.33. 

The amount of residual leave was correlated negatively with selection, 
probably because older people tended not to be selected and had more 
leave. On the other hand, very small amounts of residual leave were 
related to non- selection. Thus, of eight employees who had 8 hours or 
less of either annual or sick leave, six were not selected] and one 
of the two who were selected had excused use of leave due to automobile 
accident. 

V. INTERCORRELATIONS 



Intercorrelations among all measures for all applicants who took 

particular measures are given in Table IV-A, appendix. Results may differ fre: 

tables restricted to sets of measures, 

PART THREE - RELATIONSHIPS BETWEEN VARIOUS EVIDENCES AND FINAL RATINGS 

The job-element procedure clearly defines two aspects of the rating 
process* 

Screening applicants who are not at least 
barely acceptable 

Rating acceptable competitors in terms of potential 
for superior performance 

Part Two dealt with the screening function. This Part deals with the 
relationship between various evidences and evaluation of superiority 
among those who are at least barely acceptable. 

I. ANALYSIS OF EXTENT TO WHICH EVIDENCES AFFECTED SCREEN-OUT AND 
FINAL RATING 

Of the 5>Q applicants who completed all forms and tests, 26 passed the 
screen-out element. The 26 who passed the screen-out element were 
rated by the panel on the following five elements? Reliability and 
Dependability (screen-out)] Job Aptitude and Interest] Following 
Directions] Ability as a Member of a Team] and Dexterity and Safety. 
Relationships between the various evidences and the total credits on the 
elements given for these 26 applicants are indications of the relation 
between the various evidences and judgments of relative superior 
potential. 
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XI. difference in effect of evidences on sel ection and on rating 

* AFTER MEETING SCREEN -OUT 
A . Data 

Table A on the next page repeats from Table III the correlation 
between various evidences in the examination and selection for all 
SO applicants, and shows the correlation between the same evidences 
and number of credits on the elements for the 26 who passed the 
screen-out. 
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TABLE A 



Correlations Between Various Factors and Judgments of Rating 
Panel for Screen-Out and for Crediting After Screen-Out 
( Table IV-B at the endof this report shows comparative means 
and standard deviations) 



Correlation with No* of 



Correlation with credits after passing 
selection-N = gQ screen-out - N s 26 



Supervisor's overall rating* 

*(N is less because 1 super- 
visor didn't rate this) 


-.1780 


+.3253 


Self-rating ability 


*,5162 


+.36OO 


Self-rating, Interest 


+.5135 


+ .6)462 


Test of potential for 
reading 


+.7006 


+.5U53 


Test of potential for 
shop arithmetic 


+.538? 


+ .220)4 


Test 1QQA 


+ .L93 8 


+ M9S 2 


Test ILOOE 


+.5653 


+.5308 


Residual sick 
leave 


«,l52ii 


-.369U 


Residual annual 
* leave 


-.2162 


■ -.1693 


Years of school- 
ing 


+.U652 • 


+.3985 


Age 


-.U195 


-.2371 


Park Service 
experience 


-.2780 


-.3311 


Federal experi- 
ence 


-.2132 


-.3571 


total experi- 
ence 


-.3339 


( -.0521 



Whereas the supervisory rating had only a screen-out Effect at the 
very bottom for selection, as indicated before, it had\a positive 
effect on the number of credits for those who passed this screen-out. 
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“ The effects of years of schooling, test of potential for reading, 
and possibly the test of potential for shop arithmetic, seemed to 
have less effect on the number of credits than on selection vs, 
non-selection. Probably, certain educational factors, particularly 
in the arithmetic test, which is related only in a general way to 
the work of Gardener, tended to become exhausted after affecting the 
screen-out. On the other hand, certain indications which may have 
p een less prominent in the screen-out process tended to increase 
in importance after the "barely acceptable" factors were taken care 
of in the screen-out process. The self-rating in interest rose to 
be the factor showing highest correlation with panel judgment; .65. 
Test 100 A showed no decrease and Test 10QE showed only a slight 
decrease , 

B. Analysis of Education Effects 

The following figures may help to show what happened as an effect 
of the screen-out process. The figures show correlations between 
years of education, and the tests and self-rating, 

TABLE B 

Correlations Between Years of Education and Tests 

and Self-Ratings 



Correlation 'with Years of Correlation with Years 
Education f r r Entire of Education for Those 

Test or Self Rating Group Who Met Screen-Out 

N ° 5~Q N = 26 



100A 


.1x211 


,U807 


10 0E 


.1* 611 


.3731 


Potential for 
reading 


.6560 


.26U3 


Potential for 
shop arithmetic 


.5 266 


,2U72 


Self-Rating in 
ability 


,U?28 


.171U 


Self-Rating in 
Interest 


.U685 


.0U16 



The above table indicates that we may separate two aspects of the effects 
of education. One aspect which affected Teat 1QQA especially was not 
affected by screen-out of applicants. In fact, screening out the applicant 
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(and incidentally reducing the standard deviation of Test 100A from 
17 to 13 as shown in Table IV-B at the end of this report) seemed to 
eliminate factors not common to this test and education, thus raising 
the correlation. It will be noted Test 100A correlates very high 
with dotting tests, which Spearman had found to be among the best 
indicators of his general factor. The first aspect of education is 
that aspect which remains after the application of a screen-out, a 
screen-out such as used in the present examination. 

A second aspect of education appeared, which was affected by the screen-out. 
Whereas the first aspect was related to Test lOOA, this second aspect 
seemed to be in common with tests like Potential for reading and Potential 
for arithmetic and with self-ratings of ability and interests. The screen- 
out process, by eliminating the effect of this second aspect, reduced 
the correlation between the self-rating in interest and years of education 
from .hi to .OU. 

The second aspect of years of education seems to have been affected in 
the screen-out process. If the judgment of the rating panel can be 
accepted as satisfactorj'’, this may mean that factors such as our tests 
of potential and our self-ratings are measures of the aspect of education 
which should be considered in screen-out for certain trainee jobs. 

Again, if the judgment of the rating panel is considered to be satisfactory, 
the test of potential and self-rating blanks are still useful in providing 
ranks after screen-out. In fact, the highest correlation with the final 
panel rating for those who passed the screen— out element is .65 for the 
self -rating in interest which correlated only . Oij with the residual efi.ee t 
of education in the screened-out group (which we have termed the first 
aspect of education), ■ 

A follow up study and studies in other occupations will be conducted to 
indicate whether it is .safe to 1 ignore the first aspect of education and, 
if it is necessary to consider it, how much weight we should give to 
tests in which this aspect predominates, 

III. Intercorrelations of Various Measures Which Affect Particular 
E lement Domains 

For some of the element domains, it is possible to correlate relevant 
data from different sources; Supervisory appraisals; self-ratings ; and 
sometimes tests. 

Results are shown in Table V, at the end of this report, in three parts, 
as follows ; . * 
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Part 1: Element Domain - Aptitude and Interest. (Showing inter- 

correlations of all measures for the group who passed the screen-out.) 
Measures of aptitude and interest include: Supervisory appraisal of 

(overall) capabilities or potential. Total self-rating in ability, 

Total self-rating in interest and willingness ; Test Scores; School- 
ing) Examination rating of element Aptitude and Interest. 

Part 1 of Table V shows the intercorrelations of aU tests and self- 
ratings for those who passed the screen-out. Note that all correlations 
are for only the 26 employees who passed the screen-out* 

• The measures correlating highest with the examination rating for the 
element Aptitude and Interest are: Total Self-Rating in Interest and 

Willingness (,6?)j Test 100E — Following Oral Directions ( . 62 ) ; Total 
Self-Rating in Ability (.63)j and Test of Potential for Reading (.52), 

Part 2: Element Domain - Following Directions. Measures of Following 

Directions include: Supervisory appraisal of Following Directions, 

Self-rating of ability in items U6-57 on the Self-Rating Form (see 
below) 5 Self-Rating of Interest in these items; Total score on each 
test; and Examination Rating of Element Following Directions. 

Items U6-57 on the Self-Rating Form cover the category Carefulness, 
and seemed ncre related to Following Directions than items in other 
categories, of the form. Items it 6 to 57 are shown under the table. 

Part 2 of Table V shows the intercorrelations of the various measures. 
The inter correlations of the tests are not given since they appear 
in Part 1 of Table Vi 

Two applicants had very low self-rated ability and interest scores 
in items R6-57. In order to see the extent to which correlations 
were unduly influenced by these two cases, correlations are given 
for both the entire group who passed the screen-out, and for the group 
after eliminating the two cases: 

The highest correlation with the examination rating on the element 
Following Directions was for Test 100E — Following Oral Directions 
(.70); next were Reading Test ( .IiB) ; Test 100A — Gross Dexterity (.U7)j 
Self-Rating of Interest in items U6-57 (.36); Supervisory Appraisal-- 
Following Directions (.31); and Self-Rating of Ability in items 1:6-57 

(.2U). 

Part 3: Element Domain - Teamwork, Measures of teamwork include: 

Supervisory appraisal of working in a crew and self-rating of ability 
in three teamwork items on the selx’-rating form. These items are 
shown under the table. Tests do not seem appropriate for this element, 
and are not shown i 11 the table. The correlations with the examination 
rating for the element Teamwork were ,66 for the supervisory appraisal 
of Working in a Crew, .23 for self-rating of interest in the three 
relevant teamwork items, and . OU for self-rating of ability in the three 
teamwork items. 
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IV, DETAILED ANALYSIS OF RELATIONSHIPS BETWEEN ELEMENT RATINGS 
AND~ SUPERVISORY ' APPRAISALS 

Each element rated by ths panel corresponded to a particular category 
on the supervisory appraisal form. The 

correlation between the supervisory appraisal and the element rating 
has been given above, for three of the elements. 

Because of the importance of supervisory ratings in promotion programs, 
it is useful to note not only the correlation between the supervisory 
appraisal and the corresponding element rating but also the actual scatter- 
plot to see the way in which the relationship holds. This kind of analysis 
is made for each of the elements, in Table VI in the appendix, 

PART FOUR - INFORMATION SECURED AFTER THE EXAMINATION 

I, ANALYSIS OF NEGATIVE RELATIONSHIP BETWEEN SUPERVISORY RATINGS 
AND SELECTION; AS AFFECTED BY ASSIGNMENT 

Before the examination was held, ths rating panel had expected that 
selection would be influenced to a great extent by supervisory ratings. 

As seen in Part Two, the supervisory ratings did not have a positive 
effect on selection, aside from, the elimination of those applicants with 
the lowest supervisory appraisal. 

Informal discussions vrith raters indicated that job assignment might 
have had an effect on supervisory appraisals which was balanced out by 
the rating panel during their careful evaluation. 

' For example, a security check is made of employees assigned to sensitive 
locations. It is possible- that even though assignment of applicants is 
not made on the basis of ability, and the security check would have to 
do with eliminating the poorest employees rather than selecting the best, 
the fact that a security check is made might influence the supervisor 
to rate an employee high even though there would be no evidence support- - 
ir,g a high degree of ability* 

Among the k9 employees appraised for overall capability or potential 
by the supervisors, job assignment correlated , N 21 I 4 with supervisory 
rating but -.2188 with selection. The scatter-plots are shown in 
Table VII in the appendix. For those who met the screen-out, the correla- 
tion with job assignment was +.2B9kSor panel rating (n = 26) and + ,123ii 
for supervisory appraisal (n = 25). 



fone applicant was not given an overall supervisory appraisal. Among the 
25 who were given appraisals and who met the screen— out, the correlation 
with job assignment was ,2685 for panel rating* 
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II, MINORITY STATUS 

A, Selection 

Among the 50 competitors who completed all papers, U2 were Negro, 8 white. 
Of the U2 Negro competitors, 19 were selected. Of the 8 white competitors, 
5 were selected, 

B, Job-Element Examination Rating 

The total credits assigned by the rating panel to the 26 competitors who 
met the screen-out were as follows (cumulative percents in parentheses): 

Total 



Credits 


Negro 


White 


17 


h (1950 


. 


16 


— ' 


1 (20$) 


15 


5 ( h3 % ) 




ih 


1 US$) 


2 (60$) 


13 


2 (57$) 


1 (80$) 


12 


2 (67$) 


— 


11 


2 (76$) 


— • 


(required for) 10 


3 (90$) 


1 (100$) 


passing) 

9 


1 (95$) 




8 


— 




7 


1 (100$) 


~ 


Total 


21 


5 




18 



18 



C. Supervisory Rating of Evidences Considered by the Rating Panel 

1, Overall supervisory appraisal 

The overall supervisory appraisals of capabilities and potential 
were as follows (emulative percents in parentheses): 



Average Supervisory 
Appraisal 


Selected 


Not Selected 


Negro 


White 


Negro 


White 


U.o 






3 (135?) 


l (3356) 


3,5 


1 (656) 


— 


1 (1756) 


1 ( 6756 ) 


3.0 


10 (61%) 


2 (UC56) 


12 (7056) 


— 


2,5 


k cm) 


2 (8056) 


3 (8356) 


— 


2.1 


1 (8 9%) 


— 


— 


— - 


2.0 


2 (10056) 


1 ( 10056) 


h (10056) 


— - 


■ 1.5 


— 


— 


— 


1 (100^) 



No, Appraised 
No. Not Appraised 
Total 



18 

1 

19 



23 



23 
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2 , Total Interest score, on Aptitude-Interest Blank 

The following table is for the aptitude-interest scores (self- 
ratings) on the Aptitude-Interest Blank for all applicants who 
filled in the blank (cumulative percents in parentheses): 



Interest Score 


Negroes 


Whites 


160-16U 


1 ( 2 %) 


— 


155-159 


jt wr 1 


1 (13%) 


150-15U 


2 (WT ' 


— 


11*5-12*9 


u ww) 


— 


1U0-1UE 1 


r vuw) 1 


3 (50%T 


135-139 


2 ( 3756) 


1 (63%) 


130-132* 


— 




125-129 


1 (2*055) 


1 (75%) 


120- 122* 


3 U7%) 


1 (88%) 


115-119 


1* (56%) 


— 


110 -111* 


1 (58%) 


— 


105-109 




— 


100- iol* 


1 (60%) 


-- 


95-99 


2 (65%) 


1 (100%) 


= 90-9U 


1 (67%) 


— 








80-81* 


2 (72%) 


— 


75-79 


1 CW1 


— 


70-71*. • 


1 (77%) 




65-69 


-- 


— 


60“6~U 


2 ( 81%) 


— 


— 0^9 


8 (100%) 


— 



Total - U 5 
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3, Total ability score, on Aptitude-Interest Blank 

The following table is for the total ability scores (self-ratings) 
on the Aptitude- Interest Blank for all applicants who filled in 
the blank (cumulative percents in parentheses)*. 



Ability Score 


Negroes 


■Whites 


210-219 


2 (36) 




200-209 


— 


~ 


190-199 


l (7$) 


,1 (13%) 


18 0-189 


“2 (12%) 


— 


170-179 


— 




160-169 


u ( 21 %r 


— 


' ISO - 159 




rrr^w - 


1U0-1U9 


2 (26%) 


1 (75%) 


130-139 


STEQfT - 


nwr 


120-129 


% ($i%7~ 


— 


110-119 


1 C5W 


-- 


’ 100-109 


2 (38%) 


— 


' 90-99 


U (68%) 


— 


80-59 


2 (72%) 


— 


76-79 


~rvrm~ 


— - 


60-69 ” 


— 


1 (100%) 


“ 50-59 


1 (82%) 






-- 


— 1 


30-39 


2 (86%) 




20-29 . 


2 (91%) 


— 


10-19 


2 (96%) 


“ 


0-9 


2 (100%) 


— 



Total U3 8 



21 
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It. Test of Potential for Reading 

•The following table is for scores on the test of Potential for 
Reading for all applicants who took the test blank (cumulative 
percents in parentheses): 



Reading Test Score 


Negroes 


Whites 


20 


3 ( 7%) 


2' (20%) 


2h 


3 (lltf) 


• 1 (38%) 


23 


6 (2350 


-- 


22 


u ( 375 rr 


1 (50 %f 


21 


h { blf) ] 


1 i - T -1 \ ~ — 


20 


2 (5 W~ 


1 (63%) 


19 


l (53%1 


. 


18 


-- 


-- 


it i 


1 (56%) 


-- 


3.6 


i {sm 


l (75%) 


15 


1 (603) 


i fBBjO 


1U 


1 (63%) 


— 


13 


— 


— — 


12 


1 {65 % ) 


— 


11 


3 ( 72 %) 




10 


ttwt 




0-9 


11 ( 100;* ) 


l (ioo%7 



Total - 



U3 



8 ■ 




22 



22 



5 , Test of Potential for Arithmetic 

The following table is for scores in the Test of Potential for 
Arithmetic for all appli cants who took the test ( cumulative 
percents in parentheses)! 



Arithmetic Test Score 


Negroes 


Whites 


E5 


E (9%) 


2 (252) 


EE 


3 0-6%) 


1 (382) 


E3 


3 (23%) 


— 


hZ 


2 (282) 


— 


El 


1 (302) 


i (502) 


E0~ 


2 (352) 


1 (632) 


39 


3 ( i-2%) 




36 


3 ( E9% ) 


— 


37 


3 ism 


1 (752) 


36 


TT5BF1 


1 ( 882) 


35 


E ( 67%) 


-- 


3E 


1(70%) 




33 


— 


— 


32 


-- 


-- 


31 


1 (72%) 


— 


30 


-- 


— 


29 


2 ( 772) 


— 


28 


2 (8125 


-- 


2? 


l (8E2) 


-- 


26 


— 


-- 


25- 


1 (862) 




0-2E 


6 (1002) 


i (1002) 



Total E3 8 
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6. Test 100A - Gross Dexterity ' 

The following table is for scores in Test 100A — Gross Dexterity 
for all applicants who took the test (cumulative percents 
in parentheses): 



Score, Test 100A 


Negroes 


Whites 


120-12U 


5 (12*) 


3 (38%) 


110-119 


3 (19%) 


3 (75%) 


100-109 


3 (26 wr 1 




90-99 


8 UM) 


— s 


80-89 ' 


5 (56%) 


1 (88%J 


70-79 


3 (63%) 


— 


• 60-69 


6 (77%) 




50-59 


2 (81*)” 


1 (100?) 


UQ-U9 


2 ( 86% ) 


— 


30-39 


2 (91%) ’ 


“ “ 


20-29 


— 




10-19 


2 (95%) 


— 


0-9 


2 (100%) 





Total - 1|3 • 



8 
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7 . Test 100E - Following Oral Directions 



The following table is for scores in Test 100E— Following Oral 
Directions for all applicants who took the test (cumulative 
percents in parentheses): 



Score, Test 100E 


Negroes 


Whites 


lf?G-15>9 


5 (12%) 


6 (75%) 


1U0-1U9 


h ( 21 %) 


— 


130-139 


3 (28%) 




120-129 


6 <IZ2jn ' 




110-119 


2 (hl%) 


— 


166-109 


2 (51 JT~ 


— — 


90-99 


¥ (60%“) 


1 (08%; 


BO- 89 


3 (67%) 


— 


70-79 


2 urn 


1 (100%) 


60-69 “ 


2 (11%) 


-- 


50-59 ~ 


• 


-- 


U0-U9 


5 (88%) 




30-39 


3 (95%) 


— 


20-29 




— - 


10-19 


1 ( 98 %) 


-- 


0-9 


1 (100%) 


-- 



Total k3 '8 



O 

ERIC 



25 



25 



D , Summary 

There were too few subjects for definitive comparisons. 

However, the tendencies indicated by the results may be summarized, 
for further cheeking in additional studies. 

The point reached by approximately 75% of the white applicants may be 
a point of comparison. 

With the total Job Element rating, 5 7 % of the Negro applicants reached 
the number of credits reached by 80% of white applicants. In comparison 
for the evidences used by the rating panel t 



Evidence 


Percent of Negro and of Whites 
receiving a rating reached by 
about 75% of Whites 


Supervisory overall appraisal 
among all $0 appraised applican 


83% Negro, ^£<f a White 
:s 


Test of Potential for 
Reading, among all 51 applicant.' 


i 58% Negro, 7 5/S White 


Test of Potential for 
Arithmetic, among all 51 ap- 
plicants 


56% Negro, 75% White 


Interest Score, 
Aptitude-Interest Form, among 
all' Si applicants 


hO% Negro, 75% White 


Ability Score, 

Aptitude-Interest Form, among 
all 51 applicants 


2 6% Negro, 75% White 


Supervisory overall 
appraisal among 26 
non-selected applicants 


17% "Negro, 67% White 


Test 10QA— Gross 

Dexterity among all 51 ap- 
plicants ? c 


19% Negro, 75% White 


Test 100E- -Following 

Oral Directions, among all 5l 

applicants 


12% Negro, 75% White 




26 



26 



III. TURNOVER FIRST SIX MONTHS 

A, Withdrawal and Turnover Figures 

The agency reported general satisfaction with the selected trainees. 

However, seven applicants withdrew. Two applicants— —one a laborer and. 
one a messenger- automotive vehicle operator, had withdrawn before 
completing tests and forms. The following five applicants withdrew from 
the program after taking tests and forms, four of these after being 
selected. 

Applicant No, 1. One employee withdrew after taking the test, 
but before sele ctions were made, asking that his name be with- 
■ di-awn. He wished to continue in his assignment as tractor- 
operator because "to take a reduction in salary of over $.20 
per hour is not worth while." 

Applicant No, 2 . Before the program began, but after being 
selected, another tractor?- operator refused the assignment to 
gardener- trainee "as the monetary loss would be too great for 
my present situation." 

Applicant No, 3 . One applicant left the trainee position after 
he began the program, and returned to his f ormer position as 
laborer, because home studies proved too great a burden in the light 
of his large family and the necessity of holding a second job, 

. Applicants Nos. ii and $, Two employees left for what appeared to 
be low motivation and potential for training, and returned to 
laborer jobs# 

B. Withdrawals and Turnover, By Prior Job 

The following table on the next page shews withdrawals, examination outcomes 
and turnover for applicants in different original positions. 

The results support the hypothesis of the worker- trainee examination, that 
motivation for a new job is related to the extent to which the job offers 
a substantial increment in one's career pattern. Among U tractor and 
motor vehicle operators, 3 withdrew. Among iiO laborers, for whom the 
gardener- trainee program represented an opportunity for substantial career 
improvement, only h withdrew. One clerk- typist was selected and remained 
on the job, but he had done tree work, and had indicated motivation by the 
fact that he applied for a position outside his present work environment. 




Number of Applicants with Various Outcomes, By Original Jobs 
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C, Scores on Evidenc es, and Ratings In Examination for Those 
Who Withdrew 

1 , Scores on Evidence s 

Table VIII-A in the appendix gives the scores on the various 
evidences for the five applicants who withdrew after taking tests 
and forms, described as applicants 1 through J? in Section A above. 
Table VIII-A also shows for each of the five applicants the per- 
cent of all those selected who got higher or lower scores, on each 
evidence. 

Most significant may be Applicants Nos. h and £, whose withdrawal 
from the program seemed most related by their supervisors to lack 
of interest and aptitude. Fewer than 20% of all the selected 
applicants were lower than each of these two applicants in Supervisory 
Appraisal, Overall; Interest, on Interest- Analysis Blank; Test 100E, 
Oral Directions; Residual Annual Leave; and Park Service Experience. 
More than 70 percent of all the selected applicants were higher 
than each of these two applicants in Interest, on Interest-Analysis 
Blank ; Ability, on Interest-Analysis Blank; Test 100E — Following 
Oral Directions; Residual Sick Leave; Residual Annual Leave; and 
Park Service Experience. 

The last column shows the average difference, for each evidence, 
between the percent who got higher and the percent who got lower 
scores. This difference shows the extent to which the average for 
all five employees who withdrew was in the ■lower range of scores 
for each evidence. It is therefore an average measure of the 
relation between the evidence and later withdrawal and turnover. 

The highest differences are 60 percent for the Test of Potential in 
Arithmetic; p3% for the Test of Potential in Reading; U3 percent 
for Ability on the 'Aptitude-Interest Blank; 3h percent for Super- 
visory Appraisal, Overall; 33 percent for Residual Sick Leave; and 
27 percent for Test 100E--Foilowing Oral Directions. The difference 
for Interest on the Aptitude-Interest Blank is only 12 percent on 
the average, because Applicants 1 through 3 Quit for other motivating 
reasons than lack of interest in gardener- trainee work, discussed 
in Sections A and B, above. Applicants Nos, 3. and 2 were tractor 
operators who, although interested in the gardener items on the 
Aptitude -Interest Blank, did not perceive the gardener- trainee as a 
career field. Applicant No. 1 withdrew even before he was rated, and 
Applicant No, 2 withdrew before entering the program. Applicant No. 3 
withdrew because he needed to keep a second job, which was not 
possible with home studies required by the trainee program . It is 
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Interesting to note that Applicants Nos. 1 and 3 were relatively low in 
- self-ratings ■ of ability on the Aptitude-Interest Blank , just as were 
Applicant s = Nos. H and 5, which was one pre-indication that gardening may 
not be part of their career pattern. 

2 » Examination Ratings 

Of the seven who withdrew, three withdrew before the examination ratings 
were given . 

Table VIII-B in the Appendix gives the credits on each element and in 
the total job-element examination for the four applicants. Applicants 2 to 
5, who withdrew after being selected (described in Section A). 

Ail four who withdrew after being rated and selected received only the 
minimum credit (2 points) on the element '’Aptitude and Interest. 

(Table V7.II-B shows "0$ lower"for each applicant in this element.) 

Thus, the panel's rating of this element was in a sense substantiated 
hy later turnover* The rating of this element reflected tne panel* s 
judgment, considering all the evidences shown in Table VIj_I-A and 
discussed above under "Scores in Evidences." 

The last colirm in' Table VIII-3 is comparable to the last column in 
Table VIXI-A, The size of each percent shows the extent to which the 
applicants who withdrew are in the lower range of credits on the element. 

The average difference for- the element Aptitude and Interest is $ 0 %, 

For comparison, eliminating Applicant No. 1 in Table VIXI-A, who with- 
drew before he was rated on the elements, the highest average differences 
were : 

Evidence in Table VIXX-A 
Test, Potential for Arithmetic 
Supervisory Appraisal, Overall 
Test 100E, Following Oral Directions 
Ability, on Aptitude-Interest Blank 
Test, Potential for Reading 
Residual Sick Leave 

Interest, on Aptitude -Interest Blank 

The above figures indicate that the rating of the element Aptitude and 
Interest predicted later turnover about as well as the most predictive 
of the single evidences, but not better. 

The total examination rating included elements not related to turnover, 
as can be seen in the other entries in Table VIII-B, 



Ave, Difference for 

Applicants 2-9 
* — 

52 % 

h5% 

hM 

Ui% 

3S 

2Q% 
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Table VIII-3 - shows the number of credits in each element, and also 
the percent of those selected who got more or fewer credits. The last 
" column shows the average difference, for each element, between the 
percent who got higher and the percent who got lower credits. This 
difference shows the extent to which the four employees who quit after 
being selected were in the lower range of scores for the element* . 

It is therefore a measure of the relation between the element rating 
and later turnover. 



Most predictive of later turnover is the element rating for Aptitude 
and Interest, Next most predictive is the element Reliability and 
Dependability. The examination rating as a whole is predictive of 
turnover, but since it includes elements that are not all related 
positively, to prevention of turnover, it is less related than the first 
two elements in the table. 



IV. STUDY OF MULTIPLE- CHOICE QUESTIONS IN PART II OF APTITUDE-INTEREST BLANK 

Part II of the Aptitude-Interest Blank included six multiple- choice 
self-descriptive questions on safety record, and two on reliability. 

Examples are: (Applicants were told to circle each answer that applies, 

^ ana to fill in Job Nos. where required. ) 



What is your safety record on jobs where you used power 
equipment such as power tools, power lawn mowers, etc,? 

Circle each answer that applies. Also, fill in the 
Job Numbers if you did operate the equipment on a job. 

A. I did not operate such power equipment on a job in the 
last 3 years. 

B. In the last 2 years, on Job Mumber(s) (fill in Job Nos.) __ 
I operated such power equipment and I was not Injured 

to need medical attention. 

C. Ill the last year, on 'Job Number(s) (fill in Job, Nos.) 

•I operated such power equipment, and I was not injured, to 
need medical attention. 

D. Within the last 2 years, I operated such power equipment 
on the job, and I have been injured, to need medical 
attention. 



How often have you been late for work, or left before you were 
supposed to? 

A, Several times a week. 

B. No more than once a week . 

G» No more than once a month. 

D. No more than once in six months . 

E. Never 
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These questions were not scored before the applicants were selected, 
although the answers were available to the rating panel. The purpose 
of inclusion of these direct questions was experimental, to see 
whether such questions get meaningful answers. 



The following table shows the numbers and cumulative percents of selected 
and of non- selected applicants who gave various numbers of negative 
answers or omits to the eight questions; 



No. of Negative 
Answers or Omits 


Remained in 
Program 
N = 20 


Withdrew Initially 
After Taking Tests 
N = 2 


Quit Trainee 
Job 

N - 3 


Non-Selected 
Applicants 
Who Had. Not 
Withdrawn 
N = 26 


6-- 7 


-- 


-- 


— 


6 - 23% 


5 




-- 


H 

l 

VjJ 

U) 


2 - 31% 


h 


1 - 5% 




1 - 67% 


l - 35% 


3 


1 - 10% 


1 - 5o% 


1 - 100^ 


1 - 38% 


2 


U - 30% 


— 


— 


h -.5h% 


1* 


8-10% 


1 - 100$$ 


— 


U - 69% 


0 


6 - 100^ 






8 - 100$l 



The above table indicates that three or more negative answers or omits were 
given to the multiple- choice questions by only 10^> of those who remained 
in the program, by 38% of those who* were not selected, by one of two who 
withdrew after, taking tests, and by all three who quit after entering the 
training program. 

As predictors of turnover, the multiple- choice questions seemed to be quite 
powerful, even more so than any other evidence in the examination. It 
seemed to be as predictive of turnover as the job-element examination 
ratings for the elements Reliability and Dependability, and Dexterity and 
Safety which were the areas for which the questions were prepared. 

Table IX in the Appendix presents results for competitors Nos. 1 through S 
in the same format as Tables VIXI-A and VIXI-B. The average difference 
in the last column of Table IX, indicating the extent to which the com- 
petitors who withdrew are in the lower range of scores for the multiple- 
'choice questions (''lower" meaning more negative answers) is U7%. Omitting. 
Applicant No, 1, to be comparable to Table VIII-B, the average difference is 

62 %, 
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APPENDIX TABLES 
Promotion to Gardener-Trainee 



App. 1 



TABLE I 




The last column is "% in non-modal cells." A very high or very low reli- 
ability can ha caused by chance if almost all cases are in the cell with 
the modal frequency. For example, for ’’Potential Ability, 11 four ratings 
are 4-U, while only two ratings vary between supervisors. These two carry 
the load in determining the value of the reliability coefficient, -.n thia 
case . 6983 , which is therefore of small reliability. . Of special value are; 
absences, with non-modal percent of 30$ and reliability of *47 j follow 
directions with non-modal percent of 70.5 and reliability of ^ .45» Safety - 
Notices danger spots with U3% and «Ulj and overall ability with 91% and .62, 

TABLE II 

Regression Effect 

For some cases, only one supervisor was given. The amount of expected 
regression in estimating average ratings is as follows, based on cases 
where there were two supervisors for "Overall Value" where r was ,62; 



Overall Value, 
Superv, No. 1 


Tot. 

Freq. 


Free 

Supf 

U 


juenc 

■rv. 

3 


y wh< 
No. * 

2 1 


sre 

>. gave; 
1 


Ave, by 
Superv. No. 2 


Est. Ave, 
of 2 
raters 


4 


3 


3 




- 


- 


U.o 


U.O 


3 


30 


3 


21 


6 


* 


2,9 


2,95 


2 


10 


- 


3 


6 


1 


2.2 


2.1 



I Where only one rater rated the applicant (9 applicants), the average 

S rating used for further study was U if Supervisor No. 1 rated U* 3 if 

' Supervisor No. 1 rated 3 and 2.1 if Supervisor No, 1 rated 2« 

' O 
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TABLE III 



App, 2 



Correlations with Select vs. Non-Select, on the Basis 
of Application of Rating: Schedule in Job-Element _ 



Ex amination 

(One applicant who withdrew after completing forms and- 
taking tests was eliminated from consideration in this table) 

N 



JlafiioiL. 



Group 



r With Select c 1, 
Non-Select = 0: or 
Other Indications 



supervisory rating, overall 
abili ty 



supervisory Rating, 
Reliability' — absences 



Supervisory Rating, 
overall 



Supervisory Rating, 
s otential bility 



Supervisory Rating, 
Following. directions 



Supervisory Rating, 
Safety— at fault 



(1) All, except one selected 
applicant whose supervisor 
did not feel able to give 
overall rating 



(2) Omitting 2 applicants 
who had not themselves 
submitted forms, and whose 
non-selection was influenced 
by this ' 



All applicants 



All 



All 



All applicants 



All applicants 



SI 



h9 



-.1852 
(Negative ) 

But only one had rating 
below Satisfactory, 
and he was not selected 



-.1780 

(Negative ) 



Some effect 
rated lower 
not selected 



2 cases 



than 3» 



Some effect, 3 cases 
rated "not dependable" 
by both supervisors, not 
selected, 3 cases rated 
"not dependable" by only 
one supervisor, 2 selected, 
1 not selected 



Some effect, One rated 
lacking by both supervisors 
one by one supervisor. 
Neither selected. 



No effect— U cases rated 
"only simple directions" 
by 2 supervisors, 2 se- 
lected, 2 not selected: 

9 cases so rated by one 
supervisor; 5 selected, 
h not selected 



No effect — 3 cases: 
2 selected, 1 not 
selected 





App. 3 



TABLE III (Continued) 



Factor 


Groun I N 


r With Select s 1, 
Non-Select - 0; or 
Other Indications 


No, of negative words in com- 
ments by supervisor 


(1) All applicants 


52 


-,20?6 
(Negative ) 


(2) Omitting 2 applicants 
who did not submit 
forms 


50 


-.1827 

(Negative) 


Presence of negative 
words in comments by 
supervisor 
(Presence = lj 
Absence - 0) 


(1) AH applicants 


52 


-.2523 

(Negative) 


(2) Omitting 2 applicants 
who did not submit 
forms 


50 


-.20UU 

(Negative) 


No, of positive words in 
comments by supervisor } 
if no negative words 
appear 


(1) All applicants 


52 


-..11*22 

(Negative) 


(2) Omitting 2 applicants 
who did not submit 
forms 


50 


-.1635 

(Negative) 

— *— - ■ » ■■ ■ ■ ■ 11 1 1 


Presence of positive words 
without any negative in 
comments by supervisor 
(Presence - I; 

Absence =0) 


(1) All applicants 


52 


-.0786 

(Negative) 


(2) Omitting 2 applicants 
who did nob submit forms 


50 


-.1019 

(Negative) 


Self-Rating in ability 

(A = 0, D ° 3) 


All who filled in blanks 


50 


+.5162, 


Self-Rating in interest 
and willingness 
(L = 1] W - 1) 


All who filled in blanks 


50 


+.5135 


Test of potential for 
reading 


All who filled in blanks 
and took tests 


5o 

" VJ _■ 1 


+.7006 

- j ^ „ 


Test of potential for 
shop arithmetic 


All who filled in blanks 
and took tests 


So 


+.5387 


Tost 100A 


All who filled in blanks 
and took tests 


50 


+.1*938 


Test 10QE 


All who filled in blanks 
and took tests 


5o 


+.5653 


Possession of 
driver’s license 


All who filled in blanks 
and took tests 


50 


+.32 (based on 5 
cases with no 
license, not 
selected) 
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TABLE XII (Continued) 



App . Ii 



Factor 



Group 



N 



Amount of residual 
sick leave 



(1) All who filled in 
blanks and took tests 



So 



(2) All above except one 
person who used up sick 
leave due to auto accident 



Amount of residual 
annual leave 



All who filled in blanks 
and took tests 



U9 



r With Select - 1, 
Non-Select = 0; or 
Other Indications 



-,152U 

(Negative) 

However, effective at 
almost no leave: 



Amt- sick 
leave 


*1 


No. not 
selected 


0-8 hrs. 


6* 


U (67%) 


9-UO hrs. 


12 


U (33%) 


Hl-80 hrs. 


12 


5 (h2%) 


81 hrs. + 


(20 


13 (6$%)* 



*0ne had excused use of sick 
leave due to auto accident, 
and was selected. 

** Related to age. 



(Negative) But: 



Amt. sick 
leave 

0-8 hrs, 
9-80 hrs. 
81 hrs, + 



2U 

20 



No. not 
selected 

k 

9 

13 



(38%) 



-.2162 

(Negative) 

However, effective at almost 
no leave: 



Amt, annual 
leave 
0-8 hrs. 
9-UOhrs. 
Ul-80 hrs. 
81 hrs. + 



7 

30 



No.- not 
selected 

k (cO.A) 
2 (13%) 
2 (29%) 
18 (60%) 



*0ne had excused use of 
leave due to auto 
accident. 
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App. 5 



TABLE III (Continued) 



r With Select B 1 
Non-Select ~ 0; or 
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App. 6 



TABLE III (Continued) 

. r With Select * 1 
Non-Select = Oj or 



Q roU p N Other Indications 



r dw 

Length of experience 
in Park Service 


All who filled in 
blanks and took 
tests 


So 


-, 2?80 

(Negative) 

Yrs* exp* 

1 

1% 

2-2% 

3-5 

Sh -io 
10% + 


No, not 

f selected' 

1 1 (1002) 

6 2 (33*) 

12 6 (SO*) 

12 U (33*) 

10 5 (50*) 

9 8 (89*) 


Length of experience 
in Federal Service 


All who filled in 
blanks and took 
tests 


SO 


-.2132 

(Negative) 


Total length of 
experience 


Ail who filled in 
blanks and took 
tests , 


So 


-.3339 

(Negative ) 
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TAB L E XV- A. Inter correlations Among Measures, for Every Applicant Who Took Particular Measures. Max. liN^l 

(Two applicants who filled in no forms and took no tests, and were therefore not selected, are not 
counted for any comparison, although they were given supervisory evaluations. One applicant who withdrew 
after taking tests, and was not considered" further, was counted for the tests.) 1 



App. 7 
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TABLE IV-B 

Comparative Means and Standard Deviations for 
the Total Group of 50, and for Group of 26 
Passing Screen-Out ' 



Factor 


Total Group 


Group 

Pass. Screen-Out 


Maximum Score 


Mean 


Standard 


Mean 


Standard 


Deviation 


Deviation 


(for tests, etc.) 


Supervisor 1 s overall appraisal 
of capabilities ^ potential* 

*(N is U9 and 25> 1 supervisor 
di dn 1 1 appraise this) 


2,83 


.56U5 


2.7U 


• U032 


# 

* 

U 


Self-rating, ability 


113.88 


55.206 


1U1.27 


37.10U 


2U0 


Self-rating, interest 


100.30 


■ U3.252 


127.69 


29.900 ' 


160 


Test of potential for 
reading 


l6.SU 


7.786 


21.58 


3.128 


25 


Test of potential for 
arithmetic 


35*00 


10.180 


UQ.OU 


U.6U5 


16 


Test 100 A 


Ul.30 


17.010 


U8.00 


12.819 


12U 


Test 1Q0E 


102 . 68 


U3.88U 


127.19 


23,612 


159 


Residual sick leave 
(hours) 


131.96 


185.950 


126.39 


' 153.156 


- 


Residual annual leave 
(hours) 


126.16 


97.728 


113.77 


92.766 


- 


Years of schooling 


9.3U 


2.22U 


10.27 


1.767 


- 


Age 


35. 6U 


10.U36 


31.62 


7.277 


- 


Park Service experience 
(No, of 6 month periods) 


11. 9U 


10.736 


10.08 


9.162 


- # 


Federal experience 
(No. of 6 month periods) 


1U.98 


12.U90 


13.81 


12.500 


* 


Total experience 

(No, of 6 month periods) 


36.06 


19.U76 


29,92 


13-3U2 


- 
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Table IV- A for inter correlations for all applicants. 
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lowing Oral ! +.1506 ,■ +.I. 9 I 1 I • ; +.1^27 +.7013 

Directions ! (+.lii33) ! (+.0156) r (+.£1|62) (+. 6699 ) 

•Note that eliminating 2 competitors with very low self-rated ability and interest scores in items Ii 6-$7 
reduced the correlation of ability and interest from .73 to .22. S. D. for 26. cases were 9*20 for 
ability and 6.32 for interest; and for 2li cases were 6.5h for ability and 2.6.5 for interest. 
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App. 12 



TABLE V s PART 3 
Element Domain; Teamwork 



Group; All who passed screen-out - N = 26 



- — - 


Supervisory Appraisal, 


Exam, Rating .on 


Evidence 


Working in Crew 


Element 


Supervisory appraisal. 
Working in Crew 


— 


+.6579 


Self-rating of ability 
in 3 Teamwork items" 


- . 009 O'" - * 


+. 0 U 28 ** 


Self-rating of 
Interest in 3 




_ _ * 


Teamwork items* 


+ ..269if :H: ' 


+ ,2338" ■ 



•^1 terns are; 26, Work with others in a crew; 27. Work with a crew 
using equipment, taking care not to injure someone else; 28. Work 
carefully with others in a crew, to prevent senior, injury or death 
to one of the workers. 

■ 8 #Self -Rating on Ability ; Only 3 applicants had low scores; all received 
low supervisory appraisals and low exam rating in this domain. 

Self-Rating in Interest: Only 2 applicants had a low score —(3) 'while 
all others got the maximum score ( 6 ). The 2 with low score had low 
supervisory appraisal and low exam rating in this domain. 
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App . 13 



TABLE VI 

Relationships Between Element Ratings and Supervisory Appraisals 



Element and Appraisal 
Factor and Pearson r 


Relationship shown by scatter -plot and cumulative percentages 


Element; Reliability and 
Dependability (Screen-out) 
vs. Supervisory 


Average Supervisory 
Appraisal (Slim of 
3 phases)# 


Element Rating 


u 


3 


2 


Appraisal 

Factor; Reliability 


12 


5-56% 


7-78% 


2-25% 


and Dependability 


11.5 


2-75% 


- 


1-38% 




ii 


2-100% 


2-100% 


1-50% 




io.5 


~ 


- 


1-63% 




10 


“ 


- 


1-75% 




9.5 


- 


- 


1-88% 




9 


* 


- 


1*100% 




n 


-9 


9 


8 :■ 

1 


Group i All who passed 
screen-out ■ 

N - 26 ' 

r ■ .h632 


■s-The supervisory appraisal for this factor included throe 
phases: Unauthorized absences, unauthorized early quits, 

and overall dependability. The appraisal sheet for absences 
and for quits' included the follow. ng checklist categories; 
arbitrary values used to quantify the appraisals are shown 
in parentheses: Never (u) j no more than once in 6 months 
( 3) * no more than once a month (2); no more than once a week 
(l); more often than once a month (0). Overall dcpsndabili 
had two categories: Dependable arid reliable (It); not 

dependable and reliable (3) . Note that the values used to 
quantify the appraisals are arbitrary, and do not reflect 
the U-credit scale used by examiners to rate elements. 




To quantify the appraisals systematically, ^he highest 
category was consistently given an arbitrary value of U, 
and categories below values in descending order • ) 




In the above group, all of whom passed the screen-out 
element, only two received an overall appraisal of "not 
dependable" by any supervisor. The one whose total was 9 
was so appraised by one supervisor , his only appraiser; the 
one whose* total was 9.5 was so appraised by one supervisor, 
but was appraised as dependable by another* 
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App. lU 



TABLE Yl (Continued) 



Element: 



Screen-out, and 
Reasons for 
Screen-out 

vs. 

A ppraisal 
Factor : 

Reliability 

and 

Dependability 



Group : 

All who failed 
screen-out 



"Reliability and dependability" is the screen-out 
element, and is the only element rated for all 
applicants in the examination. Those who failed 
to be rated acceptable in this element were not 
rated further. Since this is the screen-out, all 
evidences supporting an ineligible rating were considered 
here. The standard provides that lack of potential 
ability and interest affects the screen-out, since 
reliable work demands a minimum. For example, the 
agency had advertised in the announcement that 
possession of a driver's license would be needed. 

Failure to , get a license was taken as an evidence 
of lack of ability or interest. The following scatter- 
plot shows the average supervisory appraisal for 
applicants screened out for evidences that could be 
identified by comments on the examination rating 
blanks, (Additional evidences were no doubt considered 
by the panel, but not recorded. It would not be 
practical to expect the rating panel to record all 
evidences that were considered in each case,} 



N * 2li 



(See chart on next page) 
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App . 15 



(Screen-Out Continued) 



TABLE VI (Continued) 



Reason for Screen-Out Given on' Rating Blank 



Average 
Supv e ^ 

Appraisal 


Reason 

Not 

Given 


No 

License 


No License 
No Poten, 
Ability 


No 

Poten. 

Ability 


No Poten. 
Ability, 

Poor Attend, 


Poor 

Attend, 


12 


1 


3 ( 75 %) 


1 


5 (71$) 


h UW) 


- 


11.5 


- 


- 


- 


1 (86J6) 


3* (78 36) 


1 (5<tf) 


11 


“ 


- 


- 


- 


- 


- 


10.5 




1 (1003) 


- 


1 (loot) 


- 


- 


10 


- 


- 


- 


- 


1 (89=1) 




9.5 


' ~ 


- 


“ 


- 


- 


1 V (100?) 


9 


- 


- 


- 


- 


1 " (100*) 
' — 


- 



1 u ^ 1 



1 

Sum of 3 phases of Reliability and Dependability, 

-"Rated not' dependable by both supervisors, 

*#Rated not dependable by one of two supervisors? 1 case of the 3* 
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The above scatter-plot includes Ik who received maximum appraisals 
in dependability (12) j and 10 who received less than maximum appraisals, 
Among the Hi who received maximum appraisals in dependability, 4 were 
screened out for poor attendance records, showing that supervisory 
appraisals do not always reflect accurate records. On the other hand, 
of the 10 who received less than maximum appraisals, attendance Wo.s 
mentioned as a consideration in the screening out for 7. 

This scatter-plot is a supplement to the first one in this section, 
above for persons who passed the screen-out. A comparison of botn 
scatter-plots shows that the appraisal did not differentiate those 
who passed the screen-out from those who failed, except for the 
few appraised as undependable by two supervisors, 



However, the first 



lew apursiacu SO - -- - fc j i 

scatte'r-elot showed that after the various evidences were used to 

present 0 scatter-plot" 

Ss , -.jaSd2s.*ss*isSdis “s kssk * ftset 00 ” reening 
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TABLE Vi (Continued) 



Eleme nt: Job Aptitude 

and Interest, 
vs. Supervisory 



Factor: Overall 

ilities or 
Po ten tial 



All who passed 



screen-out, except 1 
not appraised in this 
factor 



N 



25 



r - , 36 lLi 



E lenient : Fo 1 lowi ng 

Directions, 

vs. Supervisory 
Appraisal 
Fac tor ; Following 
Directions 
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The supervisory appraisal factor closest to the element Job 
Aptitude and Interest was an overall appraisal, which follows, 
with assigned values in parentheses: "From your appraisal, 

indicate in your opinion, your summary rating of the applicant's 
capabilities or potential for target position: Outstanding (h) i 

excellent ( 3 ) j satisfactory ( 2 ) j unsatisfactory ( 1 ), 

Following is the scatter-plot: 



Avt rage 
Supervisory 
Appraisal 1 


a 

1 .. _ 


Element 

. 3 

. - - 


Rating® 

2 


i 


3,5 




1 


- 


-■ 


3 


5 


U 


a 


i 


2,5 


1 


2 


3 


_ 


2 


- 


- 


u 


- 








n 


6 


7 


ii 


i 



In overall capabilities or potential 
2 . . 

In job aptituae and interest 

The correlation of ,36 is based on the fact that U of the 11 
who got 2 credits in the element were appraised by supervisors 
as lower than the others. 

Note: Only i applicant was appraised as "unsatisfactory ' 1 5 

he is not represented in the above scatter-plot because he 
was screened-out* He had also been appraised by supervisor 
as undependable. 



The categories in the supervisory appraisal for Following 
Directions and arbitrary values were: ften gets directions 

which change whut he is doing, and he does not have trouble 
following them (U ) 5 sometimes gets new directions and does not 
have trouble (3); simple directions and does not have trouble 
(2)| usually has to have simple directions repeated (1), 

Following is the sc a tier -plot; 

(See chart on next page.) 
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Continued) 



TABLE V.I( Continued ) 
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App , 18 



TABLE Vi (Continued) 



Element ; Dexterity and 
Safety, 



vs. Supervisory 
Appraisal 
Factor: Safety 

and Dexterity 



Group s All who passed 
screen-out except 1 not 
ippraised in this factor 

n E 25 



r - -.0112 



The supervisory appraisal categories for safety were not 
in a unitary scale. The highest category was really a. 
different area than the other categories. The categories 
and arbitrary values were; Has observed and reported 
dangerous or unsafe conditions in his work area (U)j Has 
never had an at- fault accident with equipment on job (3)> 

Has not had such accident in last 2 years ; Has had one 
such accident in last 2 years (l)j Has had more than one such 
accident in last 2 years (0), 

The following scatter-plot is for the average supervisory 
appraisal. (There were 2 cases where one supervisor 
appraised the applicant U for observation but 1 for safety 
record. In the examination, one was screened-out j the other 
was given a rating of 2 for the element Dexterity and Safety. 
In the other cases, no deficient safety record was appiaised 
hy a supervisor when his appraisal was for observation.) 



Average 
Supervisory 
Appraisal ' 


E; 


lenient Rating^" 




““V 


3 


! 2 


1 


U 


i 


5-5Q3 


* 


h-67% 


3*5 


- 


- 


2-253 


- 


3 


- 


3-803 


U- 753 


2-1003 


z.S 


- 


1-503 


1-883 




2 


- 


1-1003 


- 


- 


1 


- 


- 


l-ioo3 




n 


i 


10 


8 


6 . 



i 



O 

ERIC 



In Safety, Dexterity 

■K-Includes 1 case appraised U and 1 by one supervisory and 
appraised h and 3 by another supervisor. The appraisal of 
U and 1 was counted as 1.5 so the average appraisal was 2.5. 

The following scatter-plot is for the supervisory-appraised 
area of observing unsafe conditions alone, compared to the 
element rating: 



(See chart on next page.) 
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App. 19 



TABLE Vf (Continued) 



(Continued) 



Average 

Supervisory 

Appraisal 


Element Rating-- 


Dexterity and 


Safety 


a 


3 


2 


i 












Observing Sc 
reporting 
danger, noted 
by 2 super. 


i 


U-U0j6 


1-11? 


3-50% 


Observing & 
reporting 
danger* noted 
by 1 super* 




2-601 


U-5 6? 


2-83? 


No observing 
& reporting 
danger 


- 


U-loog 

— — i 


ii-100!?: 

! 


1-100? 



n ' 1 10 9 6 



Element 
Ra ting 

Total . 
Credits 
vs # 

Supervisor y 

Factor: 



^Observation of unsafe conditions 

There is no consistent relationship in the above chart* 

High degrees of observation noted in supervisory appraisals 
are associated with element ratings varying from 1 to U* 



The supervisory appraisal form concluded with the overall 
appraisal of capabilities or potential* discussed above 
in its relationship to the element Job Aptitude and Interest. 

The following scatter-plot relates this overall appraisal 
to the total credits in all the job elements in the 
examination. 



Overall capabilities 
or potential 

Group i All who passed 
screen-out and were 
appraised overall 

n - 25 



r = .3253 

(For those meeting 
passing score of 10 

JERIC 



r = .51UU) 



Supervisory 

Overall 






Total 


Credits 


for 


All 


Elements 


Appraisal of 
Potential 


17 


1 16 


i ^ 

! 15 ' 


1U 


i 

13 


12 : 


ii j; 


i 

10 : 


9 *! 7* 


3.5 


- 


- 


i i 


1 


- | 


- , | 


** ! 


- 


- j * 


— 


a 


- 


2 i 


2 ; 


2 j 


i 


i ; 


i 


1 1 1 
I 


2.5 ■ 


- 


1 


i| 


1 ; 


2 : 

I 




- j 


- 


i 

- | * 

i 


2 


— i 




1 


i 


i 

1 

i 


1 ; 
) 


1 


2 


1 



UiU3U 2.2.3 11 

(# See next page.) 



App, 20 



Element 
Rating __ 

Total 

Credits 

vs , 

Su pervisory 
Appraisa l ; 

Total of 
all items 
on Supervisory 
Appraisal 

all who passed the 
screen-out and were 
given supervisory 
evaluations on all 
fac tors 

n * 2 S' 

r = ,3203 

(For those meeting 
passing score of 10 
credits, r = .0096) 



O 




TABLE Vi (Continued) 



-* (from previous page). 

The minimum passing score was 10 . Total credits of 9 and of 7 
were failing. Competitors had to meet the passing total score 
as well as meet the screen-out. 

Although Table III showed that the overall appraisal was 
not a factor in meeting the screen-out element, and although 
the two competitors shown on previous page who met the screen- 
out but failed to get 10 credits in the element got a relative! 
high appraisal, the appraisal did have an effect on the relative 
ranks of those who passed the examination. The mean element 
credits for those appraised 3 or 3*5 and who passed the exam- 
ination ( rot 10 credits or more) was Hi. 5. The mean element 
credits for those whose appraisal was 2.5 was 13.3* Tne mean 
element credits for those whose appraisal was 2 was 10,3, 




* 7 and 9 fail 

The results for this scatter-plot are much like those for 
the previous one, showing that the relationship holds for 
applicants Who pass the examination. 
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App. 21 



TABLE VII. Relation of Job Assignment to Relationship Between Supervisory 
Appraisal of Overall Capabilities or Potential* and Selection 1 

(Frequencies and Cumulative Percents) 



Average 

Supervisory 

Rating 


S 


elect 


e d 


Not 


8 e 1 e c 


ted 


Executive 

Mansion 


N CP- 
Central 


Other 

Assignments 


Executive 

Mansion 


NCP — 
Central 


Other 

Assignments 


h 


— 


-- 


— 


2 - 67 $ 


2 - 20% 


— 


3.5 




1 - 135? 


-- 


1 - 100 $ 


1 - 30 % 


— 


3 


— — 


3 - 50$ 


9 - 60 $ 


— 


3 - 60 % 


9 - 69 $ 


2.5 




U - 100 $ 


2-11% 


— 


2 - 86 $ 


1 - 11% 


2.1 




= = 


1 - 80 $ 


— 


— 


— 


2 




— 


3 - IQOfa 


— 


2 - 100 $ 


2 - 92 $ 


1.5 




— 


— 


— 


— 


1 - 100 $ 


Total n 


0 


8 


15 


3 


10 


13 



%ith Executive Mansion valued 2, National Capitol Parks-Gentral 1* and 
Others 0* and with Selected valued 1 and Non-Selected valued 0* Pearsonian 
correlations with Assignment are +.U21U for supervisory appraisal and 
-.2188 for selection. 
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App. 22 
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Number of Negative Answers Given By Applicants Who Withdrew After Taking Tests and' Forms 

( Comparable in format to Tables VIII A and B) 



